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March  13,  2003 

MEMORANDUM  FOR  INSPECTOR  GENERAL,  NATIONAL  SECURITY  AGENCY 

SUBJECT:  Review  of  the  Quality  Control  System  at  the  National  Security  Agency 
Inspector  General  Office  of  Audits  (Report  No.  D-2003-6-005) 


We  are  providing  this  memorandum  for  your  information  and  use.  The 
Government  Auditing  Standards  require  that  audit  organizations  that  conduct  audits  in 
accordance  with  the  standards  have  an  appropriate  internal  quality  control  system  in  place 
and  undergo  an  external  quality  control  review  every  3  years  by  an  organization  not 
affiliated  with  the  organization  being  reviewed.  Our  review  of  the  National  Security 
Agency  (NS  A)  Office  of  the  Inspector  General  (OIG)  Office  of  Audits  was  to  ensure 
compliance  with  Government  Auditing  Standards  (GAS).  As  the  organization  that  has 
audit  policy  and  oversight  responsibilities  for  audits  in  the  DoD,  we  facilitated  and 
oversaw  the  conduct  of  this  external  peer  review  of  the  NSA  OIG  Office  of  Audits.  To 
avoid  unnecessary  duplication  and  disruption,  the  audit  external  quality  control  review  of 
NSA  was  done  concurrently  with  the  management  review  of  the  NSA  OIG  conducted  by 
the  Office  of  the  Assistant  Inspector  General  for  Intelligence,  Office  of  the  Inspector 
General  of  the  Department  of  Defense. 

Background.  The  foreign  intelligence  mission  of  the  NSA  is  both  national  and  defense 
in  nature,  and  it  encompasses  signals  intelligence,  information  security,  and  operations 
security.  The  Office  of  Audits  is  responsible  for  producing  quality  results  in  terms  of 
making  constructive  recommendations  to  significantly  improve  NSA  operations; 
identifying  funds  to  be  put  to  better  use;  and  preventing  and  detecting  fraud,  waste,  and 
mismanagement  in  accomplishing  the  NSA  mission.  The  Inspector  General,  NSA  is 
delegated  the  authority  from  the  Director,  NSA/Chief,  Central  Security  Service  to 
conduct  audits.  Audits  within  NSA  are  executed  by  the  Office  of  Audits  under  the 
management  and  direction  of  the  Senior  Assistant  Inspector  General  for  Audits.  The 
National  Security  Agency/Central  Security  Service  Office  of  Audits  Office  of  the 
Inspector  General  Audit  Manual  (hereafter  referred  to  as  Audit  Manual),  January  2002, 
provides  guidance  on  the  operation  of  audits  within  NSA. 

Quality  Control  Review  Objectives.  The  objectives  of  the  review  were  to  determine 
whether  the  system  of  quality  control  for  the  Office  of  Audits  in  effect  for  the  year  ended 
June  30,  2002,  was  designed  to  provide  the  NSA  OIG  with  reasonable  assurance  of 
material  compliance  with  established  policies,  procedures,  and  government  auditing 
standards  in  the  conduct  of  its  audits  and  that  the  system  of  quality  control  was  being 
complied  with  for  the  year  then  ended.  The  Office  of  Audits  issued  1 1  final  audit  reports 
during  July  2001  through  June  2002,  the  period  reviewed.  Appendix  A  contains  a 
summary  of  the  quality  control  review  process. 

Review  Results.  The  system  of  quality  control  for  the  audit  function  of  the  NSA  OIG  in 
effect  for  the  year  ended  June  30,  2002,  has  been  designed  in  accordance  with  established 
policies,  procedures,  and  government  auditing  standards.  The  Office  of  Audits  complied 
with  the  system  of  quality  control  for  the  year  then  ended  to  provide  reasonable  assurance 
of  material  compliance  with  established  policies,  procedures,  and  govermnent  auditing 
standards  in  the  conduct  of  its  audits.  Our  review  did  not  disclose  any  material 
weaknesses  in  the  system  of  quality  control  for  the  Office  of  Audits.  The  quality  control 
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system  had  many  quality  policies,  procedures,  and  practices  that  enhanced  both  audit 
effectiveness  and  efficiency  as  well  as  skilled,  competent  staff;  high  quality  audit  manual 
and  policies;  and  well  documented  working  papers  that  contained  sufficient,  competent, 
and  relevant  evidence.  We  did  make  some  observations  where  improvements  could  be 
made  in  complying  with  government  auditing  standards  and  your  internal  audit  policies 
and  procedures.  These  observations  are  not  of  sufficient  significance  to  affect  our  overall 
conclusion  as  expressed  in  this  memorandum. 

Staff  Qualifications.  The  NSA  audit  staff  is  highly  experienced  and  collectively 
possesses  adequate  proficiency  to  accomplish  the  work  assigned  as  described  in  the  Audit 
Manual.  However,  the  Office  of  Audits  should  have  a  more  balanced  training  program, 
better  documentation  of  training,  and  a  review  requirement  when  there  are  disagreements 
between  the  audit  team  and  consultants  or  internal  specialists. 

We  reviewed  training  and  educational  activity  documentation  for  the  period  of  January  1, 
2001,  through  October  15,  2002,  for  the  nine  current  full-time  auditors  on  the  NSA 
Office  of  Audits  staff.  Overall,  we  found  that  all  audit  staff  are  receiving  sufficient 
training  to  meet  the  required  continuing  education  requirements.  There  has  been  a  heavy 
emphasis  on  information  technology  course  work  to  meet  the  continuing  education 
requirements  including  the  24  hours  related  to  the  government  environment  and 
government  auditing.  Audit  staff  training  should  not  be  concentrated  on  information 
technology  for  the  24-hour  continuing  professional  education  requirement  because  it 
could  be  interpreted  as  not  being  specific  or  unique  to  the  government  environment. 

The  Qualifications  standard  requires  individuals  responsible  for  planning  or  directing  an 
audit,  conducting  substantial  portions  of  the  field  work,  or  reporting  on  the  audit  under 
GAS  to  complete  at  least  24  of  the  80  hours  of  continuing  professional  education  training 
in  subjects  directly  related  to  the  government  environment  and  to  government  auditing. 

If  the  audited  entity  operates  in  a  specific  or  unique  environment,  auditors  should  receive 
training  that  is  related  to  that  environment.  The  audit  organization  is  responsible  for 
establishing  and  implementing  a  program  for  meeting  the  continuing  education  and 
training  requirements  and  maintaining  documentation  of  the  education  and  training 
completed.  The  GAS  continuing  education  and  training  requirements  are  implemented 
through  the  Audit  Manual  Section  201.3-201.8.  The  Qualifications  standard  also 
indicates  that  an  organization  may  need  to  employ  personnel  or  hire  outside  consultants 
knowledgeable  in  such  areas  as  accounting,  statistics,  law,  engineering,  audit  design  and 
methodology,  automated  data  processing,  public  administration,  economics,  social 
sciences,  or  actuarial  science. 

The  General  Accounting  Office,  Government  Auditing  Standards,  “Interpretation  of 
Continuing  Education  and  Training  Requirements,”  April  1991  states  that  the  24-hour 
requirement  calls  for  auditors  to  obtain  24  hours  of  continuing  professional  education  in 
subjects  and  topics  directly  related  to  the  government  environment  and  to  government 
auditing  or  the  specific  or  unique  environment  in  which  the  audited  entity  operates.  The 
April  1991  Interpretation  goes  on  to  provide  guidelines  on  subjects  and  topics  that  would 
qualify  for  the  24-hour  requirement.  In  the  past,  General  Accounting  Office  personnel 
within  the  Government  Auditing  Standards  division  have  responded  to  specific  questions 
related  to  information  technology  training  as  not  qualifying  for  the  24-hour  requirement. 
The  General  Accounting  Office  personnel  emphasized  that  the  guidelines  focus  on  course 
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content  such  as  knowledge  of  government  accounting  and  auditing  standards,  legal  and 
regulatory  requirements,  and  knowledge  of  government  programs  and  activities  necessary 
for  government  auditors  to  perform  quality  audits  of  a  governmental  entity. 

Emphasis  on  infonnation  technology  training  was  caused  by  the  emphasis  on  information 
technology  systems  at  NS  A,  sponsorship  of  infonnation  technology  courses  by  the  NS  A 
at  no  cost  to  the  OIG,  and  the  low  training  budget  of  the  Office  of  Audits.  However, 
there  are  low-cost  courses  such  as  self-study  courses  on  Performance  Audits  of 
Governmental  Entities  and  Yellow  Book  Government  Auditing  Standards  that  can 
provide  audit-related  training  to  meet  the  24-hour  continuing  professional  education 
requirements  in  govermnent  environment  and  auditing.  The  Office  of  Audits  should  have 
a  more  balanced  training  program  consisting  of  continuing  professional  education 
involving  the  government  environment  and  government  auditing. 

We  also  noted  instances  where  continuing  professional  education  hours  were  not  shown 
on  the  certificates  of  completion,  and  certificates  were  not  provided.  In  some  instances, 
proof  of  class  attendance  was  based  on  memorandum  and  travel  documentation.  The 
Office  of  Audits  should  document  hours  of  continuing  professional  education  and  course 
completion  when  certificates  are  not  provided. 

In  addition,  the  Audit  Manual  does  not  contain  a  requirement  for  review  by  an 
appropriate  level  when  there  is  a  disagreement  between  an  audit  team  and  a  consultant  or 
internal  specialist. 

Independence.  The  OIG  maintains  its  independence  and  is  not  impeded  in 
accomplishing  its  intended  mission.  Our  review  was  perfonned  based  on  the  prior 
independence  standard  which  made  the  audit  organization  responsible  for  having  policies 
and  procedures  in  place  to  help  detennine  if  auditors  have  any  personal  impairments. 

The  NS  A  OIG  Office  of  Audits  met  the  GAS  independence  standard  requirements  prior 
to  January  2003,  when  the  new  independence  standard  was  effective.  See  scope 
limitations  indicated  in  Appendix  A. 

Due  Professional  Care.  The  OIG  auditors  used  sound  judgment  in  conducting 
their  audits. 

Quality  Control.  The  Office  of  Audits  did  not  consistently  use  the  prescribed 
Audit  Manual  checklists,  which  are  used  by  the  auditor-in-charge,  editor,  and  secretary  to 
process  draft  and  final  reports.  One  memorandum  report,  which  included  observations 
based  on  an  auditor’s  review  of  a  certified  public  accountant  firm’s  working  papers,  was 
not  cross-referenced  to  the  supporting  working  papers  and  lacked  evidence  of  an 
independent  referencing  review  of  the  audit  report  to  supporting  working  papers.  We 
also  identified  an  instance  where,  contrary  to  the  Audit  Manual,  the  person  performing 
the  referencing  validation  was  not  entirely  independent  because  the  individual  had  been 
part  of  the  original  audit  project  team.  Office  of  Audits  personnel  indicated  that  staff 
size,  availability,  and  impact  on  schedule  factor  into  the  selection  of  the  independent 
person.  The  Office  of  Audits  needs  to  ensure  that  persons  performing  referencing 
validation  have  no  direct  relationship  with  the  audit.  According  to  the  Senior  Assistant 
Inspector  General  for  Audits,  audits  that  ended  in  the  period  under  evaluation  used 
different  checklists  because  at  that  time  NS  A  was  going  through  a  period  of  changing 
from  one  audit  manual  to  another. 
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Audit  Planning.  The  planning  process  is  sufficient  to  ensure  that  audits 
addressing  significant  issues  are  perfonned  and  that  resources  are  efficiently  allocated  to 
complete  those  projects.  An  annual  audit  plan  is  prepared  and  issued  and  covers  major 
program  and  support  areas.  The  plan  is  reasonable  and  audits  carried  over  from  year  to 
year  are  completed  in  a  timely  manner  during  the  course  of  the  next  fiscal  year  before 
new  audits  are  begun.  The  NS  A  OIG  meets  established  criteria  for  followup  by 
maintaining  complete,  accurate,  and  reliable  records  of  the  status  of  audit  findings  and 
recommendations.  Overall,  we  found  that  the  audits  performed  were  generally  well 
planned  and  executed  with  reports  being  issued  in  a  timely  manner. 

Supervision.  In  general,  audit  supervision  at  all  levels  was  well  provided  to 
ensure  a  quality  report  acceptable  to  management.  Working  papers  of  staff  auditors  are 
reviewed  in  a  timely  manner  by  the  auditor-in-charge.  Generally,  no  one  reviewed  the 
working  papers  of  the  auditor-in-charge.  Instead  of  supervisory  review  of  auditor-in- 
charge  working  papers,  the  Senior  Assistant  Inspector  General  for  Audits  holds  monthly 
supervisory  meetings,  and  meetings  are  also  held  with  the  Deputy  Inspector  General  to 
provide  an  additional  level  of  supervision  to  the  audit  working  papers.  The  Senior 
Assistant  Inspector  General  for  Audits  indicated  that  the  structure  within  the  Office  of 
Audits  lacks  layers/hierarchical  structure,  and  their  structure  is  not  conducive  to 
supervisory  review  of  auditor-in-charge  working  papers.  The  NSA  OIG  Office  of  Audits 
had  compensating/mitigating  controls  in  place  to  ensure  they  met  the  supervision 
standard. 

Evidence  and  Working  Papers.  On  the  whole,  working  papers  provided 
sufficient,  competent,  and  relevant  evidence  to  support  audit  findings  and  conclusions. 
We  found  improper  classification  markings  either  higher  or  lower  than  should  be  on 
some  working  papers  and  binders. 

Internal  Controls.  The  audits  met  the  standards  for  reviewing  and  reporting 
internal  controls.  NSA  OIG  auditors  substantively  addressed  internal  controls  during  the 
performance  of  the  seven  audits  reviewed. 

Illegal  Acts,  Other  Noncompliance  and  Abuse.  There  were  no  indications  of 
risk  of  illegal  acts  or  other  noncompliance  for  the  reports  we  reviewed.  Auditors  did  not 
generally  use  legal  counsel  except  on  a  case-by-case  basis.  We  believe  the  Audit  Manual 
should  be  revised  to  include  a  mechanism  to  document  whether  or  not  legal  review  was 
needed  and  obtained. 

Reports  on  Audits.  Reports  were  well  received  by  management.  Findings  and 
conclusions  were  well  supported  and  documented.  Reports  were  clear,  concise,  and  met 
audit  objectives.  One  report  did  not  include  a  scope  paragraph  or  a  statement  that  the 
audit  was  conducted  in  accordance  with  generally  accepted  Government  Auditing 
Standards,  as  required  by  GAS  and  the  Audit  Manual.  One  audit  used  only  statements  of 
condition  for  the  finding  paragraph.  The  Audit  Manual  Section  755.3  requires  condition, 
criteria,  cause,  and  effect. 
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Recommendations.  To  make  the  improvements,  the  Inspector  General,  National 
Security  Agency  should  require  the  Senior  Assistant  Inspector  General  for  Audit  to: 

1 .  Develop  a  training  program  for  audit  staff  that  is  more  balanced  across 
all  functional  areas  to  meet  the  24-hour  continuing  education  and  training 
requirements  in  subjects  directly  related  to  the  government  environment  and 
government  auditing. 

2.  Develop  guidelines  for  documenting  and  recording  continuing 
professional  education  and  provide  training  to  the  Office  of  the  Inspector  General 
administrative  officer  on  what  constitutes  satisfactory  training  documentation  in 
accordance  with  Government  Auditing  Standards  and  the  General  Accounting 
Office  Interpretation  of  Continuing  Education  and  Training  Requirements.  April 
1991  (www.gao.gov  under  The  Yellow  Book,  Related  Guidance). 

3.  Expand  the  review  checklists  of  the  Audit  Manual  to  ensure  that: 

a.  The  security  classification  markings  on  the  working  papers  and 
the  binders  are  properly  classified. 

b.  Persons  performing  referencing  validation  have  no  direct 
relationship  with  the  audit. 

c.  Changes  in  report  content  adhere  to  Government  Auditing 
Standards  and  the  Audit  Manual. 

d.  Add  a  requirement  to  the  Audit  Manual  checklists  to  require  an 
indication  of  whether  or  not  legal  counsel  was  consulted. 

4.  Add  a  requirement  to  the  Audit  Manual  to  identify  the  appropriate  level 
of  review  when  there  is  a  disagreement  between  the  audit  team  and  a  consultant. 

5.  Remind  auditors  of  their  responsibility  to  check,  initial,  and  date  all 
applicable  checklists  prescribed  by  the  Audit  Manual  and  periodically  review 
working  paper  files  for  the  checklists  to  assure  conformance  with  Audit  Manual 
checklist  requirements. 

We  appreciate  the  courtesies  extended  during  the  review.  If  you  have  questions  on  this 
memorandum,  please  contact  Ms.  Carolyn  R.  Davis  at  (703)  604-8877.  See  Appendix  B 
for  the  report  distribution.  The  review  team  members  are  listed  inside  the  back  cover  of 
the  report. 


Deputy  Assistant  Inspector  General 
for  Audit  Policy  and  Oversight 
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Appendix  A.  Quality  Control  Review  Process 


Scope 


The  review  team  tested  compliance  with  the  NSA  OIG’s  system  of  quality  control  to  the 
extent  considered  appropriate.  These  tests  included  a  review  of  7  of  1 1  audit  reports 
issued  between  July  1,  2001,  and  June  30,  2002.  The  review  team  reviewed  working 
papers  for  the  selected  audits,  conducted  interviews  of  professional  and  administrative 
staff  members,  and  performed  tests  of  documentation. 

Scope  Limitations.  The  review  was  for  the  purpose  of  determining  whether  the  NSA 
OIG  internal  quality  control  system  was  designed  to  provide  reasonable  assurance  of 
material  compliance  with  established  policies,  procedures,  and  government  auditing 
standards  in  the  conduct  of  its  audits  and  was  being  complied  with  for  the  year  reviewed. 
We  conducted  our  review  in  conformance  with  standards  and  guidelines  established  by 
the  President’s  Council  on  Integrity  and  Efficiency.  The  review  would  not  necessarily 
disclose  all  weaknesses  in  the  system  of  quality  control  or  all  instances  of  lack  of 
compliance  with  it  because  our  review  was  based  on  selective  tests.  Because  there  are 
inherent  limitations  in  the  effectiveness  of  any  system  of  quality  control,  departures  from 
the  system  may  occur  and  not  be  detected. 

Projection  of  any  evaluation  of  a  system  of  quality  control  to  future  periods  is  subject  to 
the  risk  that  the  system  of  quality  control  may  become  inadequate  because  of  changes  in 
conditions  or  because  the  degree  of  compliance  with  the  policies  or  procedures  may 
deteriorate.  GAS  Amendment  No.  3,  Independence,  January  25,  2002,  requires  that  the 
audit  organization  should  have  an  internal  quality  control  system  to  help  detennine  if 
auditors  have  any  personal  impairment  to  independence  that  could  affect  their 
impartiality  or  the  appearance  of  impartiality.  Our  review  period  ended  June  30,  2002; 
However,  GAS  Answers  to  Independence  Standard  Questions,  July  2002,  indicates  that 
the  independence  standard’s  provisions  are  applicable  to  all  audits  for  periods  beginning 
on  or  after  January  1,  2003. 


Methodology 


From  October  2002  through  February  2003,  the  external  review  team  conducted  a  quality 
control  review  of  the  audit  function  for  the  Office  of  Audits  in  effect  for  the  period 
July  1,  2001,  through  June  30,  2002.  The  team  used  the  guidelines  and  checklists 
established  by  the  President’s  Council  on  Integrity  and  Efficiency  as  amended  February 
2002  to  ensure  that  the  review  was  in  conformance  with  GAS.  The  team  used  the 
President’s  Council  on  Integrity  and  Efficiency  checklist  items  to  review: 

•  Staff  Qualifications; 

•  Independence; 

•  Due  Professional  Care; 

•  Quality  Control; 
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•  Audit  Planning; 

•  Supervision; 

•  Evidence  and  Working  Papers; 

•  Internal  Controls; 

•  Illegal  Acts,  Other  Noncompliance  and  Abuse;  and, 

•  Reports  on  Audits. 

The  review  team  adjusted  the  President’s  Council  on  Integrity  and  Efficiency  guidelines 
and  checklists  as  appropriate  to  reflect  the  Office  of  Audits.  The  review  team  considered 
several  factors  in  applying  the  President’s  Council  on  Integrity  and  Efficiency  guidelines, 
such  as  the  size  of  the  Office  of  Audits,  the  degree  of  operating  autonomy  allowed,  and 
the  nature  of  work.  In  conducting  the  review,  the  review  team  reviewed  7  of  1 1  audit 
reports  issued  in  the  period  reviewed  and  associated  working  papers  for  the  reports. 


NS  A  Audit  Policies  and  Procedures 

National  Security  Agency/Central  Security  Service  Office  of  the  Inspector  General, 
Office  of  Audits,  Audit  Manual,  January  2002,  provides  guidance  on  the  operation  of 
audits  within  NSA.  Government  Auditing  Standards  published  by  the  U.S.  Comptroller 
General  are  the  criteria  guiding  auditors  in  their  work  to  ensure  quality  and  reliable  audit 
results.  Government  Auditing  Standards  require  that  the  internal  quality  control  system 
established  by  the  audit  organization  should  provide  reasonable  assurance  that  it  has 
adopted,  and  is  following,  applicable  auditing  standards  and  has  established,  and  is 
following,  adequate  audit  policies  and  procedures.  The  Department  of  Defense  (DoD) 
Internal  Audit  Manual  implements  the  Comptroller  General’s  auditing  standards  in  DoD. 

To  implement  an  internal  quality  control  system,  the  Audit  Manual  specifically  adopts 
and  expands  on  the  Comptroller  General’s  auditing  standards,  consistent  with  NSA 
authorities  for  use  by  its  auditors. 


DoD  Intelligence  Agency  Audit  External  Review  Process 

The  review  was  done  in  accordance  with  the  process  established  to  facilitate  the  external 
reviews  of  the  DoD  intelligence  agencies.  As  a  part  of  this  process,  we  established  a 
review  team  of  experienced  senior  auditors  from  the  DoD  intelligence  organizations 
except  for  the  organization  under  review.  The  OIG,  DoD  provided  training  and  oversight 
for  the  review. 
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Appendix  B.  Report  Distribution 


Other  Defense  Organizations 

Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 


Congressional  Committees  and  Subcommittees,  Chair  and  Ranking 
Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Anned  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Efficiency,  Financial  Management,  and 
Intergovernmental  Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International  Relations, 
Committee  on  Government  Reform 
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Team  Members 


The  Deputy  Assistant  Inspector  General  for  Audit  Policy  and  Oversight,  Office  of  the 
Assistant  Inspector  General  for  Inspections  and  Policy  of  the  Department  of  Defense 
prepared  this  report.  Personnel  of  the  Office  of  the  Inspector  General  of  the  Department 
of  Defense  and  other  organizations  who  contributed  to  the  report  are  listed  below. 

Carolyn  R.  Davis 
Craig  D.  Campbell 
Kenneth  Feldman 
Charles  Grauze 
Krista  S.  Gordon 


